Transferring synthetic lidar system data to real world domain for autonomous vehicle training applications

ABSTRACT

Methods and systems are disclosed for correlating synthetic LiDAR data to a real-world domain for use in training an model for use by autonomous vehicle when operating in an environment. To do this, the system will obtain a data set of synthetic LiDAR data, along with images of a real-world environment. The system will transfer the synthetic LiDAR data to a two-dimensional representation, use the two-dimensional representation and the images to train a model that a vehicle can use to operate in a real-world environment.

RELATED APPLICATIONS AND CLAIM OF PRIORITY

This patent document claims priority to, and is a continuation of, U.S.patent application Ser. No. 16/380,180, filed Apr. 10, 2019.

BACKGROUND

Autonomous vehicles require a significant amount of data about theenvironments in which they travel in order to detect and discern objectswithin the environment, as well as to predict actions of other objects,such as pedestrians and other vehicles. One system that is used tocollect such data is a light detecting and ranging (LiDAR) system. ALiDAR system illuminates an object with light and measures the lightreflected from the object with various detectors. The LiDAR systemanalyzes characteristics of the reflected light and developsthree-dimensional (3D) point clouds of data that represent 3D shapes ofobjects in and features of the environment.

3D data sets are also needed to train autonomous vehicles to develop aknowledge base of actions and reactions in the environment. However,LiDAR data can be time-consuming and expensive to collect in all areas,and the positions of objects within any environment can constantlychange.

This document describes embodiments that are directed to solving theissues described above, and/or other issues.

SUMMARY

This document describes methods of correlating synthetic LiDAR data to areal-world domain for use in training an autonomous vehicle in how tooperate in an environment. To do this, the system will obtain a data setof synthetic LiDAR data, transfer the synthetic LiDAR data to atwo-dimensional representation, use the two-dimensional representationto train a model of a real-world environment, and use the trained modelof the real-world environment to train an autonomous vehicle.

In some embodiments, obtaining the data set of synthetic LiDAR data mayinclude using a simulation application to generate the data set ofsynthetic LiDAR data by operating a virtual vehicle in a simulatedenvironment of the simulation application. Using the simulationapplication to generate the data set of synthetic LiDAR data mayinclude, as the virtual vehicle is operated in the simulatedenvironment: (i) accessing depth images generated by the simulation;(ii) extracting samples from image frames generated by the simulation;and (iii) correlating the extracted samples to the depth images toapproximate each extracted sample to a pixel in one or more of the depthimages.

In some embodiments, transferring the synthetic LiDAR data to thetwo-dimensional representation may include transferring the syntheticLiDAR data to a birds-eye view representation. In other embodiments,transferring the synthetic LiDAR data to the two-dimensionalrepresentation may include generating two-dimensional front-view images.

In some embodiments, the system may generate ground truth annotations invarious frames of synthetic LiDAR data to identify one or more objectsin the synthetic LiDAR data. This may be done to the three-dimensionalpoint cloud or to the two-dimensional images generated by transferringthe synthetic LiDAR data to a two-dimensional representation.

In some embodiments, the system may receive images captured from thereal-world environment. If so, then transferring the synthetic LiDARdata to the two-dimensional representation may include: (i) generatingtwo-dimensional front-view images from the LiDAR data; and (ii) mappingone or more of the objects in the synthetic LiDAR data to one or moreobjects in the captured images. The system also may generate groundtruth annotations in various frames of the synthetic LiDAR data toidentify one or more objects in the synthetic LiDAR data; if so, then atleast some of the front-view images may include one or more of theground truth annotations that identify one or more objects. In additionor alternatively, using the two-dimensional representation to train themodel of the real-world environment may include using the imagescaptured from the real-world environment, the two-dimensional front-viewimages from the LiDAR data, and a result of the mapping to train themodel of the real-world environment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating a method of transferring syntheticLiDAR data to a real world domain.

FIG. 2 is an example image from an autonomous vehicle operationsimulation system as may exist in the prior art.

FIG. 3 illustrates how two-dimensional bounding boxes may be added to aframe of data in a simulation system to identify objects within theframe and establish ground truth.

FIG. 4 illustrates how three-dimensional bounding boxes may be added toa frame of data in a simulation system.

FIG. 5 illustrates example elements of a LiDAR system that may be usedto collect real-world LiDAR data.

FIG. 6 illustrates example elements of a computing system that may beused to generate synthetic LiDAR data and process the simulated andreal-world LiDAR data as discussed above.

DETAILED DESCRIPTION

As used in this document, the singular forms “a,” “an,” and “the”include plural references unless the context clearly dictates otherwise.Unless defined otherwise, all technical and scientific terms used hereinhave the same meanings as commonly understood by one of ordinary skillin the art. As used in this document, the term “comprising” (or“comprises”) means “including (or includes), but not limited to.” Whenused in this document, the term “exemplary” is intended to mean “by wayof example” and is not intended to indicate that a particular exemplaryitem is preferred or required. Additional terms that are relevant tothis disclosure will be defined at the end of this description.

The present disclosure generally relates the generation and processingof synthetic LiDAR data such as may be used in connection with anautonomous vehicle or other robotic system. Synthetic LiDAR data iscurrently generated by and used in vehicle simulators that simulateoperation of a vehicle in an actual environment. In this disclosure,synthetic LiDAR data is used to train an autonomous vehicle and developa knowledge base for the vehicle that the vehicle can use to operate inthe real world.

Referring to the flowchart of FIG. 1, an electronic device that includesa processor may execute programming instructions to obtain a data set ofsynthetic LiDAR data (step 101). The synthetic LiDAR data will be a dataset representing 3D shapes in a simulated environment.

The system may draw upon an existing synthetic LiDAR data set, or it maygenerate one from an existing simulation application, such as drivingsimulators that are currently available for machine learningapplications. One such simulator is that known as CARLA, a known opensource simulation tool that is available for autonomous vehicle machinelearning applications. An example display from such a prior artsimulator is shown in FIG. 2, which illustrates a vehicle driving in asimulated environment. To obtain the simulated LiDAR data from thesimulator, an operator of the simulator may drive a virtual vehicle inthe simulator. The simulator will be programmed to collect syntheticLiDAR data from the simulated environment as the vehicle moves withinthe simulation. The electronic device may capture this synthetic LiDARdata and modify the simulated LiDAR data set to make it closer to anaccurate representation of a real-world domain.

To capture the simulated LiDAR data, the processor may access the depthimages of the simulation (e.g., two-dimensional images in which eachpixel in the image has a value indicating the depth associated with thepixel) and extract samples from four depth maps (front, left, right,rear) of multiple frames. Given a specific elevation and azimuth tosample at, the system can approximate each extracted sample to aspecific pixel in the depth image. Using the depth map in the simulationhelps the simulator produce more accurate LiDAR returns. The system maycapture data from any number of frames, with the system generating amore accurate representation the more frames that it captures.

Referring back to FIG. 1, the electronic device will also receive fromthe electronic simulator or another source identification of 2D boundingboxes and 3D bounding boxes that serve as ground truth annotations forvarious objects (such as pedestrians, other vehicles, and otherobstacles) in each frame (step 102). The bounding boxes serve toannotate the data and relate the object(s) within the bounding boxes toreal world objects. Examples of this are shown in FIG. 3, in which afirst bounding box 301 is added around a vehicle in a first frame whenthe vehicle is located a first distance from the observation point(i.e., at a depth within the frame with respect to a location of theLiDAR sensor within the simulation). A second bounding box 302 is addedaround a vehicle in a second frame when the vehicle has moved closer tothe observation point. In addition to the 2D bounding boxes shown inFIG. 3, the system may receive input and generate 3D bounding boxes asshown in FIG. 4, in which the system starts with a frame such as that onthe left and adds cuboid annotations 401, 402 to identify vehicles asshown in the frame shown on the right of FIG. 4. During operation thesimulation will attempt to predict the locations of these cuboids overtime.

While synthetic LiDAR data can extremely useful, it typically differsfrom the data returned from a real LiDAR system in the real world. Forexample, real-world LiDAR system returns include some noise (e.g.,errors) and interference patterns. Data captured from a real-world LiDARsystem also has different characteristics based on the object that it issensing (e.g., a black vehicle or a vehicle window will produce fewerreturns than the body of a light-colored vehicle). This domain mismatchbetween synthetic LiDAR data and real-world LiDAR data can limit theusefulness of synthetic LiDAR data.

Thus, returning to FIG. 1, to transfer the synthetic LiDAR data to areal world domain (i.e., to make the synthetic data provide a more realrepresentation of the real world), the system maps the synthetic LiDARpoint cloud to a different LiDAR representation, such as a birds-eyeview (BEV) representation (step 103). Because a BEV representation is a2D representation, this process may not include depth data but will onlyencode the point with the highest height (i.e., the greatest depthvalue) in each cell.

When performing the transfer of the synthetic LiDAR data to a BEVrepresentation, the system should mimic the sampling pattern of aparticular LiDAR system that will be installed on the actual vehiclethat will be trained, and the system should also mimic the samplingnoise that is introduced by different objects and surfaces. The systemmay do this by using a convolutional neural network (CNN) thatimplements a generative model, such as the generative adversarialnetwork approach that is known as the Cycle-Generative AdversarialNetwork (CycleGAN) model. The CycleGAN model is able to performtransformations without one-to-one mapping between training data insource and target domains. In the system discussed in this application,the CycleGAN model may be provided with two image sets: one set ofimages associated with the synthetic LiDAR data set and anotherassociated with real-world LiDAR data, and it may thus transform theimages in the synthetic LiDAR data set to those of the real world.

Thus, to transfer the synthetic LiDAR point cloud representation to theBEV representation (step 103), the system may capture images from thesimulator (i.e., the synthetic LiDAR data) (step 131) using methods suchas using a simulated camera of the simulation environment to capture orgenerate simulated images from the synthetic LiDAR data. The system mayalso receive—such as from data set captured by one or more camerasmounted on a real-world vehicle that includes a real-world LIDARsystem—images from a real-world environment (step (132).

The system converts the synthetic LiDAR point cloud to 2D front viewimages using equations such as:

c=└ atan 2(y,x)/Δθ]┘

r=└ atan 2(z,√{square root over (x ² +y ²)})/Δϕ┘

in which c represents the column of the image and r represents the rowof the image to which a particular LiDAR point (x, y, z) will beprojected.

This process creates two-channel images, with one channel for depth andone channel for intensity. Optionally, the system may only consider adepth channel of a specified size (for example, with values ranging from[0, 120] meters). After generating a set of front images for thesynthetic LiDAR data and a set of front view images for the real-worldenvironment, the system may use these image sets to train the CNN (suchas the CycleGAN model) (step 104). To do this, the system may extractand sample dense front view depth images from the synthetic LiDAR data.In a dense front view depth image, each (or substantially each) pixel ofthe image will be associated with a depth measurement, which is similarto what an ideal LiDAR system would produce if it could do so. While thedense images of the synthetic LiDAR data set may look less like realLiDAR samples, the dense image may have more information than thetypical real-world LiDAR image for the system to consider. To reducedownsampling artifacts in the point cloud, the system may create a dataset of synthetic, dense front view images. The system may add additionalchannels, such as semantic class, from the simulation to the dense frontview images.

Thus, a practical application of the LiDAR data conversion method is toimprove an autonomous vehicle simulation so that it will use syntheticLiDAR data that is closer to a real-world environment than it would bewithout the conversion. This helps to further reduce the amount of timethat an autonomous vehicle will need to use in the actual real world tobuild a knowledge base of actions and reactions in an environment.

FIG. 5 shows an example LiDAR system 501 as may be used in variousembodiments to collect LiDAR point clouds in a real-world environment.As shown in FIG. 5, the LiDAR system includes a scanner housing 505 witha light emitter/receiver shell portion 511 arranged around the perimeterof the housing. The shell portion 511 will be made of a material that istransparent to light, so that the system may emit light (such asinfrared light) through the shell portion 511 and receive light that isreflected back toward the shell portion. While the example of FIG. 5only shows a segment of the shell portion 511, in practice the shellportion will encircle the entire circumference of the housing 505.

The LiDAR system 501 will include a light emitter system 504 that ispositioned and configured to generate and emit pulses of light out ofthe shell portion 511 via one or more laser emitter chips or other lightemitting devices. The emitter system 504 may include any number ofindividual emitters, including for example 8 emitters, 64 emitters or128 emitters.

The LiDAR system 501 will also include a light detector 508 containing aphotodetector or array of photodetectors positioned and configured toreceive light reflected back into the system. As shown below, both theemitter system 504 and the light detector unit 508 may be positioned atangles from the front side of a mirror 502. The mirror 502 is connectedto a motor 503 that can rotate the mirror 360° so that the system mayemit and receive light through the entire circumference of the shellportion 511. As shown, the light emitted by the emitter system 504reflects off of the front of the mirror 502 before exiting the shellportion 511, and light received into the shell portion also reflects offof the front of the mirror 502 toward the light detector unit 508.However, in alternate embodiments, the mirror 502 may include a slot,the laser emitter system 504 may be located behind the mirror 502 at aposition from which it will direct light through the slot, and the motor503 may rotate the laser emitter system 504 and the mirror 502 as asingle unit. Optionally, the light detector unit 508 also may beconnected to a rotating structure of which the mirror 502 and laseremitter system 504 are a part so that it too rotates with the rotatinglight emitting/receiving structure.

One or more optical element structures 509 may be positioned in front ofthe mirror 502 to focus and direct light that is passed through theoptical element structure 509. As shown below, the system includes anoptical element structure 509 positioned in front of the mirror 502 andconnected to the rotating elements of the system so that the opticalelement structure 509 rotates with the mirror 502. Alternatively or inaddition, the optical element structure 509 may include multiple suchstructures (for example lenses and/or waveplates). Optionally, multipleoptical element structures 509 may be arranged in an array on orintegral with the shell portion 511.

Optionally, each optical element structure 509 may include a beamsplitter that separates light that the system receives from light thatthe system generates. The beam splitter may include, for example, aquarter-wave or half-wave waveplate to perform the separation and ensurethat received light is directed to the receiver unit rather than to theemitter system (which could occur without such a waveplate as theemitted light and received light should exhibit the same or similarpolarizations).

The LiDAR system will include a power unit 521 to power the lightemitter unit 504, a motor 503, and electronic components. The LiDARsystem will also include an analyzer 515 with elements such as aprocessor 522 and non-transitory computer-readable memory 523 containingprogramming instructions that are configured to enable the system toreceive data collected by the light detector unit, analyze it to measurecharacteristics of the light received, and generate information that aconnected system can use to make decisions about operating in anenvironment from which the data was collected. Optionally, the analyzer515 may be integral with the LiDAR system 501 as shown, or some or allof it may be external to the LiDAR system and communicatively connectedto the LiDAR system via a wired or wireless communication network orlink.

FIG. 6 depicts an example of internal hardware that may be included inany of the electronic components of the system, such as internalprocessing systems, external monitoring and reporting systems, or remoteservers. An electrical bus 900 serves as an information highwayinterconnecting the other illustrated components of the hardware.Processor 905 is a central processing device of the system, configuredto perform calculations and logic operations required to executeprogramming instructions. As used in this document and in the claims,the terms “processor” and “processing device” may refer to a singleprocessor or any number of processors in a set of processors thatcollectively perform a set of operations, such as a central processingunit (CPU), a graphics processing unit (GPU), a remote server, or acombination of these. Read only memory (ROM), random access memory(RAM), flash memory, hard drives and other devices capable of storingelectronic data constitute examples of memory devices 925. A memorydevice may include a single device or a collection of devices acrosswhich data and/or instructions are stored. Various embodiments of theinvention may include a computer-readable medium containing programminginstructions that are configured to cause one or more processors toperform the functions described in the context of the previous figures.

An optional display interface 930 may permit information from the bus900 to be displayed on a display device 935 in visual, graphic oralphanumeric format. An audio interface and audio output (such as aspeaker) also may be provided. Communication with external devices mayoccur using various communication devices 940 such as a wirelessantenna, an RFID tag and/or short-range or near-field communicationtransceiver, each of which may optionally communicatively connect withother components of the device via one or more communication system. Thecommunication device(s) 940 may be configured to be communicativelyconnected to a communications network, such as the Internet, a localarea network or a cellular telephone data network.

The hardware may also include a user interface sensor 945 that allowsfor receipt of data from input devices 950 such as a keyboard, a mouse,a joystick, a touchscreen, a touch pad, a remote control, a pointingdevice and/or microphone. Digital image frames also may be received froma camera 920 that can capture video and/or still images. The system alsomay receive data from a motion and/or position sensor 970 such as anaccelerometer, gyroscope or inertial measurement unit. The system alsomay receive data from a LiDAR system 960 such as that described earlierin this document.

The above-disclosed features and functions, as well as alternatives, maybe combined into many other different systems or applications. Variouscomponents may be implemented in hardware or software or embeddedsoftware. Various presently unforeseen or unanticipated alternatives,modifications, variations or improvements may be made by those skilledin the art, each of which is also intended to be encompassed by thedisclosed embodiments.

Terminology that is relevant to the disclosure provided above includes;

An “automated device” or “robotic device” refers to an electronic devicethat includes a processor, programming instructions, and one or morecomponents that based on commands from the processor can perform atleast some operations or tasks with minimal or no human intervention.For example, an automated device may perform one or more automaticfunctions or function sets. Examples of such operations, functions ortasks may include without, limitation, navigation, transportation,driving, delivering, loading, unloading, medical-related processes,construction-related processes, and/or the like. Example automateddevices may include, without limitation, autonomous vehicles, drones andother autonomous robotic devices.

An “electronic device” or a “computing device” refers to a device thatincludes a processor and memory. Each device may have its own processorand/or memory, or the processor and/or memory may be shared with otherdevices as in a virtual machine or container arrangement. The memorywill contain or receive programming instructions that, when executed bythe processor, cause the electronic device to perform one or moreoperations according to the programming instructions.

The terms “memory,” “memory device,” “data store,” “data storagefacility” and the like each refer to a non-transitory device on whichcomputer-readable data, programming instructions or both are stored.Except where specifically stated otherwise, the terms “memory,” “memorydevice,” “data store,” “data storage facility” and the like are intendedto include single device embodiments, embodiments in which multiplememory devices together or collectively store a set of data orinstructions, as well as individual sectors within such devices.

The terms “processor” and “processing device” refer to a hardwarecomponent of an electronic device that is configured to executeprogramming instructions. Except where specifically stated otherwise,the singular term “processor” or “processing device” is intended toinclude both single-processing device embodiments and embodiments inwhich multiple processing devices together or collectively perform aprocess.

The term “vehicle” refers to any moving form of conveyance that iscapable of carrying either one or more human occupants and/or cargo andis powered by any form of energy. The term “vehicle” includes, but isnot limited to, cars, trucks, vans, trains, autonomous vehicles,aircraft, aerial drones and the like. An “autonomous vehicle” is avehicle having a processor, programming instructions and drivetraincomponents that are controllable by the processor without requiring ahuman operator. An autonomous vehicle may be fully autonomous in that itdoes not require a human operator for most or all driving conditions andfunctions, or it may be semi-autonomous in that a human operator may berequired in certain conditions or for certain operations, or that ahuman operator may override the vehicle's autonomous system and may takecontrol of the vehicle.

The term “execution flow” refers to a sequence of functions that are tobe performed in a particular order. A function refers to one or moreoperational instructions that cause a system to perform one or moreactions. In various embodiments, an execution flow may pertain to theoperation of an automated device. For example, with respect to anautonomous vehicle, a particular execution flow may be executed by thevehicle in a certain situation such as, for example, when the vehicle isstopped at a red stop light that has just turned green. For instance,this execution flow may include the functions of determining that thelight is green, determining whether there are any obstacles in front ofor in proximity to the vehicle and, only if the light is green and noobstacles exist, accelerating. When a subsystem of an automated devicefails to perform a function in an execution flow, or when it performs afunction out of order in sequence, the error may indicate that a faulthas occurred or that another issue exists with respect to the executionflow.

An “automated device monitoring system” is a set of hardware that iscommunicatively and/or electrically connected to various components(such as sensors) of an automated device to collect status oroperational parameter values from those components. An automated devicemonitoring system may include or be connected to a data logging devicethat includes a data input (such as a wireless receiver) that isconfigured to receive device operation data directly or indirectly fromthe device's components. The monitoring system also may include aprocessor, a transmitter and a memory with programming instructions. Amonitoring system may include a transmitter for transmitting commandsand/or data to external electronic devices and/or remote servers. Invarious embodiments, a monitoring system may be embedded or integralwith the automated device's other computing system components, or it maybe a separate device that is in communication with one or more otherlocal systems, such as, for example in the context of an autonomousvehicle, and on-board diagnostics system.

As used in this document, the term “light” means electromagneticradiation associated with optical frequencies, e.g., ultraviolet,visible, infrared and terahertz radiation. Example emitters of lightinclude laser emitters and other emitters that emit converged light. Inthis document, the term “emitter” will be used to refer to an emitter oflight, such as a laser emitter that emits infrared light.

In this document, when terms such “first” and “second” are used tomodify a noun, such use is simply intended to distinguish one item fromanother, and is not intended to require a sequential order unlessspecifically stated. In addition, terms of relative position such as“vertical” and “horizontal”, or “front” and “rear”, when used, areintended to be relative to each other and need not be absolute, and onlyrefer to one possible position of the device associated with those termsdepending on the device's orientation.

The features and functions described above, as well as alternatives, maybe combined into many other different systems or applications. Variousalternatives, modifications, variations or improvements may be made bythose skilled in the art, each of which is also intended to beencompassed by the disclosed embodiments.

1. A method of correlating synthetic LiDAR data to a real-world domainfor an autonomous vehicle, the method comprising, by a processingdevice: obtaining a data set of synthetic LiDAR data; obtaining imagesof a real-world environment; transferring the synthetic LiDAR data to atwo-dimensional representation; using the two-dimensional representationand the images of the real-world environment to train a model of thereal-world environment that an autonomous vehicle can use to operate inthe real-world environment.
 2. The method of claim 1, wherein using thetwo-dimensional representation to train the model of the real-worldenvironment and the images of the real-world environment comprisesinputting two-dimensional front view images generated from the syntheticLiDAR data into a convolutional neural network.
 3. The method of claim 2further comprising generating the two-dimensional front view images fromthe synthetic LiDAR data by extracting front view depth images from thesynthetic LiDAR data, in which substantially each pixel of each frontview depth image is associated with a corresponding depth measurement.4. The method of claim 1, wherein obtaining the data set of syntheticLiDAR data comprises using a simulation application to generate the dataset of synthetic LiDAR data by operating a virtual vehicle in asimulated environment of the simulation application.
 5. The method ofclaim 4, wherein using the simulation application to generate the dataset of synthetic LiDAR data comprises, as the virtual vehicle isoperated in the simulated environment: accessing a plurality of depthimages generated by the simulation; extracting samples from a pluralityof image frames generated by the simulation; and correlating theextracted samples to the depth images to approximate each extractedsample to a pixel in one or more of the depth images.
 6. The method ofclaim 1, wherein transferring the synthetic LiDAR data to thetwo-dimensional representation comprises transferring the syntheticLiDAR data to a birds-eye view representation.
 7. The method of claim 1,further comprising generating ground truth annotations in a plurality offrames of synthetic LiDAR data to identify one or more objects in thesynthetic LiDAR data.
 8. The method of claim 1, wherein: the methodfurther comprises receiving a plurality of images captured from thereal-world environment; and transferring the synthetic LiDAR data to thetwo-dimensional representation comprises: generating a plurality oftwo-dimensional front-view images from the LiDAR data, and mapping oneor more of the objects in the synthetic LiDAR data to one or moreobjects in the plurality of images.
 9. The method of claim 8, wherein:the method further comprises generating ground truth annotations in aplurality of frames of the data set of synthetic LiDAR data to identifyone or more objects in the synthetic LiDAR data; and wherein at leastsome of the front-view images comprise one or more of the ground truthannotations that identify one or more objects.
 10. The method of claim 8wherein using the two-dimensional representation to train the model ofthe real-world environment comprises using the plurality of imagescaptured from the real-world environment, the two-dimensional front-viewimages from the LiDAR data, and a result of the mapping to train themodel of the real-world environment.
 11. A system for correlatingsynthetic LiDAR data to a real-world domain for an autonomous vehicle,the system comprising: a processor; a non-transitory memory that storesa data set of synthetic LiDAR data; and a non-transitory memorycontaining programming instructions that are configured to cause theprocessor to: access the data set of synthetic LiDAR data and extractsynthetic LiDAR data from the data set, transfer the extracted syntheticLiDAR data to a two-dimensional representation, obtain images of areal-world environment; use the two-dimensional representation and theimages of the real-world environment to train a model of the real-worldenvironment that an autonomous vehicle can use to operate in thereal-world environment.
 12. The system of claim 11, wherein theinstructions to use the two-dimensional representation and the images ofthe real-world environment to train the model of the real-worldenvironment comprise instructions to input two-dimensional front viewimages generated from the synthetic LiDAR data into a convolutionalneural network.
 13. The system of claim 12 further comprisinginstructions to generate the two-dimensional front view images from thesynthetic LiDAR data by extracting front view depth images from thesynthetic LiDAR data, in which substantially each pixel of each frontview depth image is associated with a corresponding depth measurement.14. The system of claim 11, further comprising additional programminginstructions that are configured to cause a processor to implement asimulation application that generates the data set of synthetic LiDARdata by operating a virtual vehicle in a simulated environment of thesimulation application.
 15. The system of claim 14, wherein theprogramming instructions to implement the simulation application thatgenerates the data set of synthetic LiDAR data comprise instructions to,as the virtual vehicle is operated in the simulated environment: accessa plurality of depth images generated by the simulation; extract samplesfrom a plurality of image frames generated by the simulation; andcorrelate the extracted samples to the depth images to approximate eachextracted sample to a pixel in one or more of the depth images.
 16. Thesystem of claim 11, wherein the programming instructions to transfer thesynthetic LiDAR data to the two-dimensional representation comprisesinstructions to transfer the synthetic LiDAR data to a birds-eye viewrepresentation.
 17. The system of claim 11, further comprisingadditional programming instructions that are configured to cause theprocessor to generate ground truth annotations in a plurality of framesof synthetic LiDAR data to identify one or more objects in the syntheticLiDAR data.
 18. The system of claim 11: further comprising additionalprogramming instructions that are configured to cause the system toreceive a plurality of images captured from the real-world environment;and wherein the instructions to transfer the synthetic LiDAR data to thetwo-dimensional representation comprise instructions to: generate aplurality of two-dimensional front-view images from the LiDAR data, andmap one or more of the objects in the synthetic LiDAR data to one ormore objects in the plurality of images.
 19. The system of claim 18:further comprising additional programming instructions that areconfigured to cause the processor to generate ground truth annotationsin a plurality of frames of the data set of synthetic LiDAR data toidentify one or more objects in the synthetic LiDAR data; and wherein atleast some of the front-view images comprise one or more of the groundtruth annotations that identify one or more objects.
 20. The system ofclaim 18, wherein the instructions to use the two-dimensionalrepresentation to train the model of the real-world environment compriseinstructions to use the plurality of images captured from the real-worldenvironment, the two-dimensional front-view images from the LiDAR data,and a result of the mapping to train the model of the real-worldenvironment.
 21. A non-transitory computer-readable medium containingprogramming instructions for correlating synthetic LiDAR data to areal-world domain and training an autonomous vehicle, the instructionscomprising instructions to: receive a plurality of images captured froma real-world environment; implement a simulation application thatgenerates a data set of synthetic LiDAR data by operating a virtualvehicle in a simulated environment of the simulation application;extract synthetic LiDAR data from the data set; transfer the extractedsynthetic LiDAR data to a two-dimensional representation by: generatinga plurality of two-dimensional front-view images from the LiDAR data,and mapping one or more of the objects in the synthetic LiDAR data toone or more objects in the plurality of images; use the two-dimensionalrepresentation to train a model of the real-world environment.